Pattern recognition of sensor data is currently one of the most important fields of study. The ability to classify a pattern in noisy data and then recognize that same pattern in different noisy data is a challenging problem. Applications for this type of technology are widespread. Examples include financial modeling, fitness trackers, meteorological studies, detection of failure modes in mechanical systems, human gesture analysis, audio analysis, and many more as is known by those with skill in the relevant art.
Physical fitness tracking devices on the market today suffer from numerous limitations. For example, available devices have limitations on the types of activities that they will recognize: some of the most successful activity tracking devices currently on the market are limited to simple pre-programmed activities, e.g., sitting, walking and running. The analytics for even these pre-programmed activities are inaccurate: the software comes in a one-size-fits-all solution that does not fit all users/activities, because it does not take into account individual variations based on weight, speed and real-time movement.
The forms of computing devices are changing rapidly and so too are means for interfacing with said computing devices. In the past the mouse and keyboard were the industry standard for input means however advanced technologies allow for novel possibilities. Voice input and gesture input are two newer methods for establishing a human-machine interface (HMI). It is important to be able to train computing devices to learn recognize unique gestures and voice commands and responsively execute desired functionality. For example, a voice command for launching a web browser such as “open web browser” could be used instead of or alongside traditional techniques (e.g., a mouse double click of an application icon). Additionally, physical gestures may be used to perform various tasks as well.
In today's age there exists a significant reliance on mechanical systems. Mechanical systems can be found in numerous varied industries. For example, many automated assembly lines utilize industrial robots, a type of mechanical system, to aid a manufacturing process. Furthermore, engines, motors, and turbines are pervasive technologies found in many high-risk use cases, such as those in the aerospace, energy production, and transportation industries. The reliance on mechanical systems is of the upmost importance and malfunctioning equipment must be identified as quickly as possible to ensure the safety of the general public. Many of these systems utilize sensors to detect malfunctioning equipment.
A further limitation of presently available devices includes a lack of a self-training feature: these devices are inaccurate and not useful to a large user base, because the currently supported activities can't be personalized for every user, and does not allow functionality of these devices to expand to cover new user-defined activities, e.g., weightlifting, aerobic routines, mechanical system failure modes, and custom gesture inputs. Although this limitation is widely recognized, no device on the market today has a self-training feature for user-defined activities.
Still further, the current devices have poor performance: even for the limited sets of activities that the current devices can ostensibly track, the performance may not be very robust. In real-time demonstrations of these devices, missed detection and mis-classification of activities is always an issue.
In a recent study described in an MIT Technology Review of the top three activity devices a user reported wearing three devices simultaneously and one recorded him taking 9,725 steps, the second device detected 11,981 steps, and the third recorded 6,785 steps. See R. Metz. Fitness trackers still need to work out kinks. MIT Technology Review, 27 Nov. 2013.
Specifically with respect to health and fitness applications, in open literature, there are several papers on activity recognition devices and software, e.g., Bao and Intelle's early work [2]; thesis-work by Tapia at MIT [3]; mobile sensing platform (MSP) developed at Intel Research [4]; and the various tools and frameworks for activity recognition such as eWatch [5], ActiServ [6], COSAR [7], Vigilante [8], Centinela [9], and others [10, 11, 12, 13, 14, 15, 16].
The general approach used by many prior technologies consists of three steps: (a) Identification of a ‘feature vector’ for a given (a priori fixed) set of activities; (b) Offline training of a model for this feature vector based on sensor data from a single or multiple users; and (c) Online recognition of the activity based on online computation of features from the sensor data and its comparison against the trained model. The main problems with this approach are as follows:
1. Feature selection: The performance of these algorithms critically depends upon selection of ‘good’ features. Selecting features from sensor data is not an easy task. The usefulness of any given feature is dependent in a highly nonlinear manner on the activity, on the sensor, on the placement of the sensor on the body and possibly also on the subject [17, 18]. For walking, a popular approach is to use ‘peaks’ in acceleration data as a feature vector [1, 19]; other types of time-domain [20, 21, 22, 23] and frequency-domain [2, 24] features are also discussed in literature. The number of features, however, can quickly grow in size—as many as 600 features are available in MSP [4].
The need to select a small number of ‘good’ features in an a priori manner is the main reason why existing solutions are limited to pre-programmed activities.
2. Large training data set requirement: For a given set of features, training a good model (e.g., decision tree [25, 5, 2]) requires a large amount of data. In practice, one collects a rich enough data set from multiple users to train these models [26, 9, 20, 8, 12]. This is a costly and time-intensive process. With data from just a single user, the resulting model can be fragile (e.g., sensitive to changes in frequency of the activity). With a large number of users, the model-fit to any particular user may be poor.
3. Accuracy: Once the models have been trained, the classification is typically the easiest step. Popular methods such as template matching using e.g., nearest neighbor classifiers (used by MotionX; cf., [13, 20]) and the decision tree classifier (an industry standard; cf., [25, 5, 2, 11, 8, 22, 27]) can be run very efficiently. Other popular algorithmic approaches include: instance based learning (IBL) approaches [25, 5, 28]; Bayesian networks (BN) [29, 9, 30] and Naive Bayes (NB) classifiers [8, 3, 14, 31]; support vector machines (SVM) [32, 15, 21, 33]; fuzzy If-Then rules [10, 24, 6] and artificial neural networks (ANN) [29, 12, 34].
However, these approaches suffer from the issue of robustness: the conditions for online data and the training data must be closely matched (in terms of number of sensors, window length of data, features, even subjects) for the results to be accurate; cf., [35, 5, 36, 25, 24, 37]. Many classifier approaches such as IBL and ANN can be non-robust and do not provide information on activity counts or duration, etc. [27, 33, 34].
In open literature, probabilistic approaches such as regression methods [7, 9], hidden Markov models (HMM) [38, 14] and multi-graph based techniques [39, 16] have also been considered, but their practical application suffer from the curse of dimensionality, e.g., results reported in [39, 16] lead to a graph size of 16,875 nodes making it computationally prohibitive. A simple four state HMM with five independent sensor measurements requires a total of 50 parameters that need to be learned (a specific example of this appears in [40]). This requires a large amount of training data to learn (identify) the parameters, and the problem is known to be non-convex with multiple local minima for parameters; cf., [41].
Accordingly, there is a need for methods and systems for recognition of sensor data patterns.
Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of embodiments of the present invention.
The apparatus and method components have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments of the present invention so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.